GenAI-OCR: Intelligent Receipt Processor

Receipt processing is an essential yet challenging task in fields like accounting, logistics, and retail. From extracting item details to formatting structured data, the variability of receipts makes automation a complex problem. To address this, I developed GenAI-OCR: Intelligent Receipt Processor, a project that combines the power of Generative AI (GenAI) models through OpenRouter and uses Command-R as a decision engine for robust output validation.

This project demonstrates how cutting-edge AI models can collaborate to automate workflows that traditionally required manual effort. In this article, I’ll discuss the technical implementation, the role of Generative AI in the pipeline, and the models used to bring this project to life.

What is Generative AI (GenAI)?

Generative AI refers to models designed to create or generate content—whether it’s text, images, audio, or structured data—based on input instructions. In this project:

• OCR models like PixTral-12B, Qwen-2V, and LLaMA-3.2 generate structured LaTeX tables from receipt images.
• Command-R analyzes the outputs of these models and generates the best result by evaluating their accuracy and completeness.

The project combines these generative AI models to automate receipt processing with high accuracy, showcasing the potential of GenAI in solving real-world problems.

What is OpenRouter?

OpenRouter is a platform that simplifies access to multiple AI models through a unified API. Instead of juggling multiple APIs and configurations, developers can use OpenRouter to:

• Access state-of-the-art models for tasks like OCR and decision-making.
• Integrate models seamlessly into workflows.

For this project, OpenRouter connected me to PixTral-12B, Qwen-2V, and LLaMA-3.2 for OCR tasks, as well as Command-R for decision-making.

The Technology Stack

• OCR Models via OpenRouter: PixTral-12B, Qwen-2V, LLaMA-3.2
• Decision Engine: Command-R
• Generative AI Workflow: Combining multiple generative models via OpenRouter and Command-R

How the System Works

1. Input: Receipt Image

The user uploads a receipt image, which is encoded as a base64 string and sent to the OpenRouter API.

2. Data Extraction: OCR Models

Each OCR model processes the receipt and generates a LaTeX table containing:

• Item Code: Unique identifier for each item.
• Item Name: Product name as listed on the receipt.
• Item Price: Price of each item.
• Total Price: Final amount at the bottom of the receipt.

3. Validation: Command-R

Command-R evaluates the LaTeX tables based on:

• Logical consistency: Do the items and prices add up correctly?
• Structural accuracy: Is the LaTeX code properly formatted?
• Completeness: Are all items represented?

4. Output: Final LaTeX Table

Command-R returns the best table, ensuring the output is accurate and usable.

Prompts: Crafting Precision

OCR Model Prompt

To ensure accurate extraction, the following prompt was designed:

    Extract the following information from this receipt:
    1. Item Code: A unique code for each item.
    2. Item Name: The name of the item.
    3. Item Price: The price of the item.
    4. Total Price: The final price at the bottom of the receipt.
    
    Organize the data into a LaTeX table with headers: `Item Code`, `Item Name`, `Item Price`, and `Total Price`.
    Return only the LaTeX code. Do not include any explanations or extra text.
        

Command-R Prompt

Command-R was tasked with evaluating the outputs using this prompt:

    You are a decision engine called Command-R.
    
    Your task is to analyze multiple LaTeX tables extracted from a receipt and select the most accurate and complete one.
    
    ### Receipt Item Details:
    - Item Code: Unique identifier.
    - Item Name: Product name.
    - Item Price: Price in the format `X.XX EUR`.
    
    ### Criteria for Selection:
    1. Logical accuracy: Do the items and totals match the receipt?
    2. Structural correctness: Is the LaTeX code valid and properly formatted?
    3. Completeness: Are all items and the total price included?
    
    Return only the LaTeX code for the best table. Do not add any explanations.
        

Challenges and Solutions

1. Variability in Receipt Formats

Different receipts have varying structures and languages. Using multiple OCR models via OpenRouter provided flexibility to handle diverse cases.

2. Ensuring Accuracy

OCR models occasionally made mistakes. Command-R added a critical layer of validation, ensuring only the most accurate results were returned.

3. Prompt Engineering

Creating clear and specific prompts was essential for guiding the models to generate reliable outputs.

Why GenAI?

This project highlights the transformative potential of Generative AI:

• Automating Complex Workflows: From data extraction to decision-making, GenAI models replaced manual processes.
• Collaboration Between Models: OpenRouter enabled seamless integration of OCR and decision-making models.
• High Accuracy: Combining multiple models with Command-R validation ensured robust results.

Conclusion

GenAI-OCR: Intelligent Receipt Processor demonstrates how developers can use Generative AI to automate and enhance workflows. By leveraging OpenRouter for model integration and Command-R for decision-making, this project achieves high accuracy and reliability in receipt processing.

If you’d like to explore the code, check out the GitHub repository. 💻 Also access the full article Here:Google Drive 📝 Let me know your thoughts and ideas for future improvements!

Back